Adding the option to disable the DNS processor failure or success cache by mjmbischoff · Pull Request #44932 · elastic/beats

mjmbischoff · 2025-06-19T14:20:57Z

Proposed commit message

Adds the option to disable the success and failure cache.

Motivation

This is to enable use cases that require capturing the current point in time dns record regardless of cache or ttl of the record. Such as the case of monitoring the dns server, or with recorded events that need to capture the current state of the environment. TTL captures the time frame over which the old value might be used over the current DNS record, in other words the frame time in which the agent might observe the old or new record based upon whenever the previous request was made. This unpredictability can be undesired when optimizing time-to-intervention.

Disabling the cache will have throughput implications, serial processing an event will be greater than DNS roundtrip time. For example if round-trip time to perform an DNS request is 1 ms, max throughput it limited to 1000/sec. Known use cases have are low throughput requirements. Parallelization, by for example deploying multiple agents, can be used to stretch this number. We would urge to reevaluate the use case and the use of the cache at this point.

NOTE: setting the ttl on the failure cache to 1ns achieves a similar, but imperfect effect.
NOTE: setting the ttl on the success cache is a valid option as per code, it is however ignored as also document in the code. in the documentation it is omitted as an option. Honoring setting and the ttl (min(ttl, dns_record_ttl)) is a different route. Similar to other dns client behaviour.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

non known, the default values leave the old behavior intact and the setting to trigger the new behavior is added in this PR

How to test this PR locally

Define the DNS processor, observe cache stats / resolver requests.

Related issues

https://github.com/elastic/enhancements/issues/24988

github-actions · 2025-06-19T14:21:07Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-06-19T14:21:33Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mjmbischoff? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

- QF1008, while I disagree with removing the additional qualification as it makes things more readable, removing the qualifier to appease the linter god.

andrewkroh

Can you please add to the proposed commit message to explain the why part and what is the use case for cache disablement.

beats/.github/PULL_REQUEST_TEMPLATE.md

Line 19 in c203b82

- WHY: the rationale/motivation for the changes

Turning off the caching will significantly limit the throughput of the pipeline. Even if each request takes 1ms to complete, that means the maximum throughput is 1000 EPS.

Also, the documentation for the processor will need updated to include the new configuration parameter.

mjmbischoff · 2025-06-21T09:22:25Z

Can you please add to the proposed commit message to explain the why part and what is the use case for cache disablement.

beats/.github/PULL_REQUEST_TEMPLATE.md

Line 19 in c203b82

- WHY: the rationale/motivation for the changes

Turning off the caching will significantly limit the throughput of the pipeline. Even if each request takes 1ms to complete, that means the maximum throughput is 1000 EPS.

Also, the documentation for the processor will need updated to include the new configuration parameter.

Added motivation.

~~TODO: documentation~~ Added documentation c5de66a ab103b2 e1f60c9

- document Enabled settings - Notes with warnings on throughput and compounding effects

colleenmcginnis

Some minor suggestions below.

Co-authored-by: Colleen McGinnis <colleen.j.mcginnis@gmail.com>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

github-actions · 2025-06-26T18:58:41Z

@Mergifyio backport 9.0 9.1

mergify · 2025-06-26T18:59:14Z

backport 9.0 9.1

✅ Backports have been created

Details

#45077 [9.0](backport #44932) Adding the option to disable the DNS processor failure or success cache has been created for branch 9.0 but encountered conflicts
#45078 [9.1](backport #44932) Adding the option to disable the DNS processor failure or success cache has been created for branch 9.1

This enables use cases that require resolving the current DNS record, regardless of the record's TTL or any previously cached values. It is useful, for example, when monitoring a DNS server or when recorded events must capture the environment's state at a specific moment. When a cache is used, the TTL determines the time frame in which an agent might observe a stale record instead of the current one. This unpredictability can be undesirable when optimizing for rapid time-to-intervention. Disabling the cache has significant throughput implications. The processing time for a single event will be at least the DNS round-trip time. For example, if a DNS request takes 1 ms, the maximum serial throughput is limited to 1000 events/sec. Known use cases for this feature have low throughput requirements. Throughput can be increased by deploying multiple, parallel agents. NOTE: Setting the failure cache TTL to a very low value (e.g., 1ns) achieves a similar, but imperfect, effect. NOTE: While the config allows setting a TTL on the success cache, this option is currently ignored. A future enhancement could honor this setting (e.g., by using min(configured_ttl, record_ttl)), which would align with the behavior of other DNS clients. (cherry picked from commit eee15e7) # Conflicts: # libbeat/processors/dns/dns_test.go

This enables use cases that require resolving the current DNS record, regardless of the record's TTL or any previously cached values. It is useful, for example, when monitoring a DNS server or when recorded events must capture the environment's state at a specific moment. When a cache is used, the TTL determines the time frame in which an agent might observe a stale record instead of the current one. This unpredictability can be undesirable when optimizing for rapid time-to-intervention. Disabling the cache has significant throughput implications. The processing time for a single event will be at least the DNS round-trip time. For example, if a DNS request takes 1 ms, the maximum serial throughput is limited to 1000 events/sec. Known use cases for this feature have low throughput requirements. Throughput can be increased by deploying multiple, parallel agents. NOTE: Setting the failure cache TTL to a very low value (e.g., 1ns) achieves a similar, but imperfect, effect. NOTE: While the config allows setting a TTL on the success cache, this option is currently ignored. A future enhancement could honor this setting (e.g., by using min(configured_ttl, record_ttl)), which would align with the behavior of other DNS clients. (cherry picked from commit eee15e7)

… (#45078) This enables use cases that require resolving the current DNS record, regardless of the record's TTL or any previously cached values. It is useful, for example, when monitoring a DNS server or when recorded events must capture the environment's state at a specific moment. When a cache is used, the TTL determines the time frame in which an agent might observe a stale record instead of the current one. This unpredictability can be undesirable when optimizing for rapid time-to-intervention. Disabling the cache has significant throughput implications. The processing time for a single event will be at least the DNS round-trip time. For example, if a DNS request takes 1 ms, the maximum serial throughput is limited to 1000 events/sec. Known use cases for this feature have low throughput requirements. Throughput can be increased by deploying multiple, parallel agents. NOTE: Setting the failure cache TTL to a very low value (e.g., 1ns) achieves a similar, but imperfect, effect. NOTE: While the config allows setting a TTL on the success cache, this option is currently ignored. A future enhancement could honor this setting (e.g., by using min(configured_ttl, record_ttl)), which would align with the behavior of other DNS clients. (cherry picked from commit eee15e7) Co-authored-by: Michael Bischoff <mjmbischoff@controplex.com> Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com>

Adding the option to disable the cache

c24afc9

mjmbischoff requested a review from a team as a code owner June 19, 2025 14:20

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 19, 2025

mergify Bot assigned mjmbischoff Jun 19, 2025

mjmbischoff added the enhancement label Jun 19, 2025

Merge branch 'main' into dns-cache

66db3da

mjmbischoff changed the title ~~Adding the option to disable the DNS failure or success cache~~ Adding the option to disable the DNS processor failure or success cache Jun 19, 2025

mjmbischoff added 2 commits June 20, 2025 08:55

G115: integer overflow conversion int64 -> uint32 (gosec)

9bb53da

- improved test

2d2e2a3

- QF1008, while I disagree with removing the additional qualification as it makes things more readable, removing the qualifier to appease the linter god.

andrewkroh reviewed Jun 20, 2025

View reviewed changes

Comment thread libbeat/processors/dns/config.go Outdated

Flipping disabled to enabled

c44ef87

mjmbischoff added 2 commits June 21, 2025 11:26

Merge branch 'main' into dns-cache

072a181

Adding documentation

c5de66a

- document Enabled settings - Notes with warnings on throughput and compounding effects

mjmbischoff requested a review from a team as a code owner June 21, 2025 09:58

mjmbischoff added 2 commits June 21, 2025 12:07

Adding documentation - forgot the other beats

ab103b2

- document Enabled settings - Notes with warnings on throughput and compounding effects

Adding documentation - closing note block

e1f60c9

colleenmcginnis approved these changes Jun 23, 2025

View reviewed changes

Comment thread docs/reference/auditbeat/processor-dns.md Outdated

Comment thread docs/reference/auditbeat/processor-dns.md Outdated

Comment thread docs/reference/auditbeat/processor-dns.md Outdated

Comment thread docs/reference/auditbeat/processor-dns.md Outdated

mjmbischoff and others added 2 commits June 24, 2025 08:52

Documentation: processing suggestions / fixes

28dc245

Co-authored-by: Colleen McGinnis <colleen.j.mcginnis@gmail.com>

Merge branch 'main' into dns-cache

48dde05

andrewkroh reviewed Jun 25, 2025

View reviewed changes

Comment thread libbeat/processors/dns/config.go Outdated

Comment thread libbeat/processors/dns/cache.go Outdated

Comment thread libbeat/processors/dns/dns_test.go Outdated

mjmbischoff and others added 5 commits June 25, 2025 20:31

Merge branch 'elastic:main' into dns-cache

ed01f58

Removing unused default annotation

e8159ca

Improving code doc

ba51c01

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

reverting the addition of test - didn't add more coverage.

5bfe692

Merge branch 'main' into dns-cache

228229f

mjmbischoff removed enhancement needs_team Indicates that the issue/PR needs a Team:* label labels Jun 25, 2025

mjmbischoff added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches and removed enhancement needs_team Indicates that the issue/PR needs a Team:* label labels Jun 25, 2025

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 25, 2025

andrewkroh approved these changes Jun 25, 2025

View reviewed changes

Merge branch 'main' into dns-cache

7ed24cf

andrewkroh merged commit eee15e7 into elastic:main Jun 26, 2025
203 checks passed

andrewkroh added enhancement libbeat :Processors labels Jun 26, 2025

This was referenced Jun 26, 2025

[9.0](backport #44932) Adding the option to disable the DNS processor failure or success cache #45077

Closed

[9.1](backport #44932) Adding the option to disable the DNS processor failure or success cache #45078

Merged

mjmbischoff deleted the dns-cache branch June 26, 2025 21:24

khushijain21 mentioned this pull request Jul 15, 2025

"Unblock CI: Update kafka dependency openjdk11 to openjdk17 #45381

Closed

6 tasks

vishaangelova mentioned this pull request Jul 28, 2025

[Docs] Add applies_to to DNS processor docs #45580

Merged

mergify Bot mentioned this pull request Jul 28, 2025

[9.1](backport #45580) [Docs] Add applies_to to DNS processor docs #45602

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the option to disable the DNS processor failure or success cache#44932

Adding the option to disable the DNS processor failure or success cache#44932
andrewkroh merged 17 commits intoelastic:mainfrom
mjmbischoff:dns-cache

mjmbischoff commented Jun 19, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 19, 2025

Uh oh!

mergify Bot commented Jun 19, 2025

Uh oh!

andrewkroh left a comment •

edited

Loading

Uh oh!

Uh oh!

mjmbischoff commented Jun 21, 2025 •

edited

Loading

Uh oh!

colleenmcginnis left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2025

Uh oh!

mergify Bot commented Jun 26, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mjmbischoff commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Motivation

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

github-actions Bot commented Jun 19, 2025

🤖 GitHub comments

Uh oh!

mergify Bot commented Jun 19, 2025

Uh oh!

andrewkroh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mjmbischoff commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colleenmcginnis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2025

Uh oh!

mergify Bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mjmbischoff commented Jun 19, 2025 •

edited

Loading

andrewkroh left a comment •

edited

Loading

mjmbischoff commented Jun 21, 2025 •

edited

Loading

mergify Bot commented Jun 26, 2025 •

edited

Loading